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I. 


INTRODUCTION 


This  is  a  follow-on  thesis  to  the  Kirkpatrick  and  Rushing  (2016)  thesis  regarding 
development  of  MOEs  and  MOPs  for  the  Unmanned  Tactical  Autonomous  Control 
and  Collaboration  (UTACC)  program.  UTACC  is  currently  a  complex  robotic 
program,  and  the  authors  focused  on  the  system-agnostic  capabilities  that  are  required 
for  effective  communication  between  Marines  and  machines.  As  Kirkpatrick  and 
Rushing  (2016)  state,  “It  is  a  concept  that  has  the  potential  to  change  the  relationship  of 
man  and  machine  on  the  battlefield  forever.  The  concept  employs  a  team  of  aerial  and 
ground  robots,  in  conjunction  with  complex  software  enabling  their  interaction  and 
sensor  information  exchange,  to  work  as  semi-autonomous  teammates  with  a  small 
Marine  Corps  unit”  (Kirkpatrick  &  Rushing,  2016,  p.  1). 

Just  as  in  the  Kirkpatrick  and  Rushing  (2016)  thesis,  the  authors  focused  on 
previous  UTACC  theses  to  serve  as  the  foundation  and  starting  point  into  researching 
autonomy  and  robotics  in  war  through  the  lens  of  the  man-machine  team.  According  to 
Chen  and  Bames  (2014)  and  reiterated  by  Kirkpatrick  and  Rushing  (2016),  the  two  main 
types  of  interaction  between  man  and  machine  can  be  classified  as  either  “on  the  loop” 
(OTL)  or  “in  the  loop”  (ITL).  OTL  interaction  is  the  preferred  method  of  interaction 
because  it  meets  the  UTACC  requirement  of  decreasing  the  cognitive  load  on  the  human. 
Kirkpatrick  and  Rushing  (2016)  created  quantifiable  metrics  to  determine  how  well 
UTACC  supports  “on  the  loop”  interaction.  The  authors  took  this  research  and  applied  it 
to  the  man-machine  communication  problem  set. 

Following  in  Kirkpatrick  and  Rushing’s  (2016)  MOE  and  MOP  work,  this  thesis 
recommends  MOEs  and  MOPs  to  evaluate  the  communications  modalities  required  to 
support  man-machine  teaming  in  a  way  that  supports  “man  on  the  loop”  interaction.  This 
will  support  the  process  towards  maturing  UTACC  for  eventual  use  by  the  warfighter. 
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A.  VISION  OF  UTACC 

As  the  UTACC  program  matures,  the  Marine  Corps  Warfighting  Laboratory 
integrates  the  program  into  its  campaign  of  experimentation  (COE).  MCWL  has  used 
testing  ranging  from  limited  technical  assessments  to  developmental  testing  to  evaluate 
new  functionality  in  the  program.  In  addition  to  directly  testing,  MCWL  also  sponsors 
events  for  academia  and  industry  to  participate  and  drive  innovation.  Each  of  these  events 
generates  new  data  sets  that  can  be  used  to  further  the  UTACC  program  and  assess  its 
likelihood  of  long-term  success,  justify  funding,  and  continue  research  projects.  The 
UTACC  end  state  is  incorporation  into  a  program  of  record  throughout  the  USMC. 

The  ability  of  the  Marine  Corps  to  not  only  wage  war  but  also  conduct  many  other 
missions  will  drastically  change  with  the  integration  of  UTACC  as  an  autonomous 
member  of  the  fire  team.  A  key  benefit  to  integrating  an  autonomous  robot  is  their  ability 
to  do  the  “dull,  dangerous  and  dirty”  (Singer,  2009,  p.  63)  jobs  that  are  currently 
conducted  by  humans.  Not  only  could  these  jobs  be  performed  by  robots,  but  working  in 
a  collaborative  environment  with  robots  would  allow  humans  to  leverage  the  machine’s 
capabilities  that  exceed  the  limits  associated  with  a  single  human.  Moreover,  a  single 
human  could  operate  numerous  robots  thereby  using  the  robots  as  a  swarm  and  removing 
humans  from  the  most  dangerous  portions  of  the  battlefield  (Jameson,  Franke,  Szczerba, 
&  Stockdale,  2005,  p.  2). 

B.  NECESSITY  OF  MOP/MOE 

When  research  focus  is  lost  and  the  warfighter’s  input  is  minimized,  programs 
tend  to  morph  into  something  that  is  “pretty”  or  “gold  plated”  instead  of  a  program  that  is 
useful.  To  ensure  the  success  of  any  developmental  program,  accurate,  well  defined 
MOEs  and  MOPs  are  crucial.  The  creation  of  MOEs  and  MOPs  helps  keep  programs  on 
track  throughout  their  development.  Designed  specifically  for  this  purpose,  “The 
assessment  process  uses  MOPs  to  evaluate  task  performance  and  MOEs  to  determine 
progress  of  operations  toward  achieving  objectives  and  ultimately  the  end  state”  (U.S. 
Joint  Chiefs  of  Staff  [USJCS]  J-7,  2011,  p.  ix). 
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C.  THESIS  IMPACT  AND  ORGANIZATION 

The  authors  focused  their  research  on  a  very  specific  piece  of  the  UTACC 
program:  human-machine  interface  (HMI)  MOEs  and  MOPs.  Narrow  in  its  scope,  this 
thesis  focuses  on  the  ability  for  Marine-machine  interaction  to  take  place,  a  crucial  and 
necessary  problem  that  requires  resolution  for  UTACC  to  succeed.  Developing  MOEs 
and  MOPs  to  evaluate  the  various  sensors  and  communication  interfaces  is  one  of  the 
first  steps  in  selecting  the  right  technology.  Once  solved,  these  HMI  MOEs  and  MOPs 
will  lay  the  framework  for  the  development  of  additional  UTACC  MOEs  and  MOPs  as 
the  program  evolves. 

This  thesis  consists  of  five  chapters.  Chapter  I  introduces  the  thesis  and  the 
purpose  behind  the  research  efforts,  including  the  Marine  Corps  Warfighting  Lab’s  vision 
for  the  UTACC  program  and  the  impact  this  research  will  have  on  the  Marine  writ  larger. 
Chapter  II,  the  literature  review,  explores  several  fundamental  publications  and 
documents  as  they  pertain  to  developing  UTACC  MOEs  and  MOPs.  Those  publications 
and  documents  include  United  States  Marine  Corps  Missions,  Doctrine  and  tactics, 
techniques,  and  procedures  (TTPs),  Marine/Machine  Integration,  the  principle  of 
observability,  predictability,  and  directability  (OPD)  as  it  pertains  to  autonomous 
systems,  current  communication  interfaces  and  finally  the  HMI  MOPs  and  MOEs  as  they 
relate  to  the  military  technology  selection  process. 

Chapter  III,  Research  Methodology  and  Related  Factors,  details  the  HMI  MOPs 
and  MOEs  selection  process.  After  an  overview  of  the  basic  systems  engineering  process, 
the  authors  present  UTACC  definitions,  assumptions,  constraints,  and  the  role  USMC 
doctrine  and  TTPs  play  into  the  MOE  and  MOP  process.  Finally,  the  layers  of  analytical 
development  are  described,  laying  the  framework  for  the  construction  of  the  HMI  MOE 
and  MOPs. 

Chapter  IV,  UTACC  HMI  MOPs  and  MOEs,  is  the  heart  of  the  thesis.  In  this 
chapter,  the  research  is  separated  by  components  that  are  presented  in  a  top-down 
approach.  The  recommended  modifications  to  the  MCTL  organization  are  addressed, 
followed  by  the  MOPs  and  finally  the  MOEs.  Lastly,  this  thesis  recommends  future 
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testing  scenario  metrics  and  environments  for  MCWL  to  use  in  order  to  validate  our 
research  as  well  as  further  leverage  the  results  as  the  UTACC  program  continues  to 
evolve. 

Chapter  V  is  a  summary  of  our  results  with  recommended  future  research.  As  is 
the  case  with  previous  UTACC  theses,  the  HMI  MOE/MOP  thesis  serves  as  another  link 
in  the  chain  in  the  evolution  of  UTACC  and,  therefore,  includes  recommendations  meant 
to  further  the  HMI  components  of  the  system. 

D.  CHAPTER  CONCLUSION 

UTACC  is  not  simply  employing  robots  on  the  battlefield;  it  is  employing  robots 
to  revolutionize  warfighting.  Whereas  previous  theses  discussed  the  vision,  concept  of 
operations,  and  overall  MOEs  and  MOPs  for  the  program,  this  thesis  specifically  targets 
the  MOEs  and  MOPs  required  which  ensure  that  the  man  and  machine  can  communicate 
effectively  to  complete  the  mission. 
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II.  LITERATURE  REVIEW 


This  is  the  eighth  thesis  supporting  the  development  of  UTACC  conducted 
through  the  Naval  Postgraduate  School  (NPS).  Previous  theses  discussed  collaborative 
autonomy,  robotics,  human-machine  interaction,  MOEs  and  MOPs  and  USMC  doctrine. 
Because  the  previous  theses  conducted  such  in  depth  reviews,  this  thesis  will  cover 
MOEs  and  MOPs  specific  to  the  human-machine  interface  while  citing  previous  works  as 
references.  This  literature  review  serves  to  summarize  existing  publications,  current 
work,  and  thought  processes  relevant  to  UTACC  design,  including  autonomy,  doctrine, 
communication  modalities,  MOEs/MOPs,  and  man-machine  integration. 

A.  USMC  MISSIONS,  DOCTRINE  AND  TTPS 

For  UTACC  systems  to  be  successful,  they  must  effectively  integrate  into  and 
improve  the  capabilities  of  a  Marine  unit’s  ability  to  accomplish  its  respective  mission. 
However,  there  are  no  Marine  Corps  Doctrinal  Publications  (MCDPs)  which  explain  how 
to  integrate  UTACC  systems  into  the  force. 

MCDP  1  states  that  “a  significant  advantage  can  be  gained  by  being  first  to 
exploit  a  development  in  the  art  and  science  of  war”  (USMC,  2011,  p.  17).  As  mentioned 
in  Rice,  Chhabra,  &  Keim  (2015)  and  paraphrased  by  Kirkpatrick  and  Rushing  (2016), 
Expeditionary  Force  21  (EF21)  is  the  USMC’s  vision  document  how  the  Marine  Corps  of 
the  21st  century  should  operate.  EF21  states  that  the  modern  force  will  “preserve  the 
quantitative  edge  over  opponents”  and  exploit  “innovative  concepts  and  approaches” 
(USMC,  2014).  Moreover,  EF21  also  states  that  the  Marine  Corps  of  the  21st  Century  will 
be  “light  enough  for  rapid  response”  which  is  supported  through  Jameson,  et  al.’s  (2005) 
research  as  interpreted  by  the  UTACC  program  (USMC,  2014,  p.  7).  Most  recently,  the 
Marine  Corps  released  the  Marine  Corps  Operating  Concept,  which  explicitly  drives  the 
Marine  Corps  to  exploit  automation  and  “integrate  robotic  autonomous  systems  with 
manned  platforms  and  Marines”  (United  States  Marine  Corps,  2016,  p.  16).  UTACC  is 
exactly  the  program  that  can  bridge  the  current  gap  in  automation  and  integration. 


5 


Rice  et  al.  state,  “A  mature  UTACC  system  requires  full  integration  of 
warfighting  functions  (intelligence,  maneuver,  fires,  logistics,  force  protection,  command 
and  control)”  (Rice  et  al.,  2015,  p.  17).  Kirkpatrick  and  Rushing  (2016)  developed  MOEs 
and  MOPs  for  the  UTACC  system  writ  large,  however  for  the  Human/System  Integration 
to  be  effective,  it  is  vital  to  develop  additional  MOEs  and  MOPs  specific  to  the 
communication  between  Marines  and  the  autonomous  systems.  This  thesis,  paired  with 
Kirkpatrick  and  Rushing’s  work,  will  aid  in  the  creation  of  new  doctrine  inclusive  of  the 
autonomous  systems  in  line  with  both  EF21  and  the  Operating  Concept. 

B.  MARINE-MACHINE  INTEGRATION 

Prior  research  teams  conducted  in-depth  reviews  of  Marine-machine  integration 
and  requirements  for  successful  systems.  Of  significance  to  our  focus  on  communication 
interfaces,  Kirkpatrick  and  Rushing  state: 

The  UTACC  system  will  need  to  facilitate  dynamic  information  exchange. 

Gold  (2009)  describes  the  nature  of  complex  information  exchange  in  the 
four  areas  of  “robot  to  human,  environment  to  robot,  human  to  robot, 
robot  to  environment”  (Gold,  2009).  In  addition  to  these,  UTACC 
planning  would  necessitate  the  inclusion  of  robot-to-robot  information 
exchange,  as  the  design  incorporates  more  complex  and  multiple  robotic 
systems.  Sensors  and  computers  organic  to  the  robot  systems  will  allow 
them  to  interact  with  the  environment  around  them,  but  the  UTACC 
collaborative  concept  will  require  these  robots  communicating  this  sensor 
data  to  the  other  UTACC  elements  involved  in  the  mission  including  both 
human  and  machine  teammate  elements.  It  will  therefore  be  necessary  to 
ensure  this  communication  piece  is  designed  to  present  the  sensor  data  to 
the  decision  maker  in  an  effectively  and  timely  manner.  This  subsequently 
facilitates  his  mental  picture  of  the  real  environment  around  him  and 
informs  decision-making  (Shattuck  &  Lewis  Miller,  2006,  p.  3). 
(Kirkpatrick  &  Rushing,  2016,  pp.  9-10) 

The  Army  Research  Lab  in  Maryland  describes  the  challenges  of  communicating 
with  and  integrating  autonomous  systems: 

A  critical  challenge  of  the  mid-21st  century  will  involve  successfully 
managing  and  integrating  the  collections,  teams,  and  swarms  of  robots  that 
would  act  independently  or  collaboratively  as  they  undertook  a  variety  of 
missions  including  the  management  and  protection  of  communications  and 
information  networks  and  the  provision  of  decision-quality  information  to 
humans.  Success  in  this  aspect  of  command  and  control  would  depend 
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upon  developing  new  C2  concepts  and  approaches,  in  particular, 
developing  and  fielding  an  effective  hybrid  cognitive  architecture  that 
leverages  the  strengths  of  artificial  intelligence  and  human  intelligence  to 
go  along  with  the  development  of  new  robotic,  communications, 
information,  and  systems  technologies.  From  the  various  observations  of 
workshop  participants,  the  traditional  balance  between  offense  and 
defense  may  shift  as  it  becomes  more  difficult  for  the  defense  to  keep  up. 

(Kott  et  al.,  2015,  p.  23) 

To  help  alleviate  the  challenges  of  communicating  with  and  integrating 
autonomous  systems,  Donald  Norman  and  Stephen  Draper  presented  the  User  Centered 
System  Design  (UCD)  concept  as  depicted  in  Figure  1.  Through  their  design  process,  the 
user  remains  the  central  focus  at  each  stage  of  development  by  asking  questions  such  as, 
“What  are  the  goals  and  desires  of  the  user,  what  tools  do  they  need,  what  type  of  task  are 
they  required  to  accomplish,  and  what  methods  do  they  prefer?”  (Norman  &  Draper, 
1986,  p.  2).  With  these  questions  in  mind,  the  UCD  process  lays  out  four  steps:  specify 
the  context  of  use;  specify  the  requirements;  create  design  solutions;  and  evaluate  the 
designs  (U.S.  Department  of  Health  and  Human  Services,  2015).  This  framework  enables 
the  designer  to  ensure  the  system  being  developed  remains  focused  on  the  needs  of  the 
user  within  the  context  of  its  operating  environment. 
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Identify  Need 


Specify  Context 
of  Use 


System 

Satisfies 


Specify 

Requirements 


Produce  Design 
Solutions 


Figure  1.  Four-Step  UCD  Process.  Source:  U.S.  Department  of  Health  and 

Human  Services  (2015). 


The  Department  of  Defense  (DOD)  published  a  report  titled  The  Role  of 
Autonomy  in  DOD  Systems  in  2012  that  discussed  the  capabilities  of  integrating  the 
autonomous  systems  in  order  to  reduce  cognitive  load  on  the  operator  while 
simultaneously  maximizing  strengths  of  the  machines: 

With  proper  design  of  bounded  autonomous  capabilities,  unmanned 
systems  can  also  reduce  the  high  cognitive  load  currently  placed  on 
operators/supervisors.  Moreover,  increased  autonomy  can  enable  humans 
to  delegate  those  tasks  that  are  more  effectively  done  by  computer, 
including  synchronizing  activities  between  multiple  unmanned  systems, 
software  agents  and  warfighters — thus  freeing  humans  to  focus  on  more 
complex  decision  making.  (DOD,  2012,  p.  1) 

Each  of  these  reports,  when  combined  and  viewed  through  the  lens  of  the  Rice  et 
al.’s  definition  of  “collaborative  autonomy,”  describe  the  challenges  to  humans’ 
operating  systems  and  the  need  for  specific  MOEs  and  MOPs  relating  to  the  Marine- 
machine  integration  to  help  shape  doctrine. 
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C.  OBSERVABILITY,  PREDICTABILITY,  DIRECTABILITY 

The  following  quote  from  Johnson’s  (2014)  work  clarifies  what  it  means  to  be 
observable,  predictable  and  directable: 

Observability  means  making  pertinent  aspects  of  one’s  status  and 
knowledge  of  the  team,  task  and  environment  observable  to  others. 
Observability  also  involves  the  ability  to  observe  and  interpret  pertinent 
signals.  It  plays  a  role  in  many  teamwork  patterns  e.g.,  monitoring 
progress  and  providing  backup  behavior. 

Predictability  means  one’s  actions  should  be  predictable  enough  that 
others  can  reasonably  rely  on  them  when  considering  their  own  actions. 
Predictability  also  involves  considering  other’s  actions  when  developing 
one’s  own.  It  is  essential  to  many  teamwork  patterns  such  as 
synchronizing  actions  and  achieving  efficiency  in  team  performance. 

Directability  means  one’s  ability  to  direct  the  behavior  of  others  and 
complementarity  by  directed  by  others.  It  includes  explicit  commands 
such  as  task  allocation  and  role  assignment  as  well  as  subtler  influences, 
such  as  providing  guidance  or  suggestions  or  even  providing  salient 
information  that  is  anticipated  to  alter  behavior,  such  as  a  warning. 
Teamwork  patterns  that  involve  directability  include  such  things  as 
requesting  assistance  and  querying  for  input  during  decision  making. 

By  using  the  OPD  framework  as  a  guide,  a  designer  can  identify  the 
requirements  for  teamwork  based  on  which  interdependence  relationships 
the  designer  chooses  to  support.  The  framework  can  help  a  designer 
answer  questions  such  as  ‘What  information  needs  to  be  shared,’  ‘Who 
needs  to  share  with  whom,’  and  ‘When  is  it  relevant.’  The  goal  of  the 
designer  is  to  attain  sufficient  OPD  to  support  the  necessary 
interdependent  relationships.  (Johnson,  2014,  pp.  68-70) 

This  OPD  framework  shifts  the  focus  from  one  individual  component,  either  the 
robot  or  the  human,  to  the  team  components  and  how  they  both  affect  one  another 
(Johnson,  2014).  Traum,  Rickel,  Gratch,  and  Marsella  (2003)  use  three  categories  to 
discuss  the  relationship  between  machines  and  humans:  agents  in  supporting  individual 
team  members,  agents  supporting  the  team  as  a  whole,  and  agents  as  an  equal  team 
member  (Traum  et  al.,  2003).  The  UTACC  program  falls  in  the  third  category  of 
assuming  an  equal  role  as  the  other  team  members.  As  noted  by  the  National  Research 
Council: 
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This  is  the  hardest  role  for  a  software  agent  to  assume,  since  it  is  difficult 
to  create  a  software  agent  that  is  as  effective  as  a  human  at  both  task 
performance  and  teamwork  skills.  Instead  of  merely  assisting  human  team 
members,  the  software  agents  can  assume  equal  roles  in  the  team, 
sometimes  replacing  missing  human  team  members.  It  can  be  challenging 
to  develop  software  agents  of  comparable  competency  with  human 
performers  unless  the  task  is  relatively  simple.  (National  Research 
Council,  2014,  p.  53) 

Through  answering  the  framework’s  questions,  the  Marine  Corps  will  be  more 
able  to  adapt  future  doctrine  to  integrate  the  effectiveness  and  performance  of  the 
communication  and  interaction  between  the  system  and  human. 

D.  COMMUNICATION  INTERFACES 

In  viewing  the  UTACC  problem  set,  redundant  communication  interfaces  are 
essential  to  achieve  mission  accomplishment  in  the  wide-ranging  tactical  environment. 
Marines  currently  use  three  sensory  modes  in  order  to  communicate  amongst  themselves 
at  the  fire  team  level:  visual,  audio,  and  haptic.  By  adding  a  machine  to  the  fire  team, 
electromagnetic  communication  also  becomes  a  viable  interface.  The  selection  of  the 
correct  mode  or  modes  of  communication  directly  relies  on  the  environment  in  which  the 
team  is  operating.  When  noise  discipline  is  required,  audio  communication  is  a  last  resort, 
but  visual,  haptic,  and  electromagnetic  are  all  viable  modes.  Conversely,  when  noise 
discipline  is  no  longer  a  constraint,  audio  communication  may  be  the  most  efficient  way 
to  disseminate  orders  and  directions  to  team  members.  The  remainder  of  this  section  will 
provide  an  overview  of  the  available  interfaces  and  is  discussed  in  greater  depth  in  later 
chapters. 

Visual  communication  is  the  primary  method  by  which  fire  teams  communicate. 
Additionally,  the  work  of  Calinon,  Evrard,  Gribovskaya,  Billard,  and  Kheddar  shows  that 
robots,  through  observation  of  human  behavior,  can  learn  collaborative  manipulation 
tasks  (Calinon  et  al.,  2009).  This  would  allow  the  fire  team  to  create  team  specific  signals 
as  well  as  enable  the  robot  to  relay  and  replicate  hand-arm  signals,  creating  a  feedback 
loop.  Some  of  the  limitations  of  visual  communications  are  poor  visibility,  restricted 
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terrain,  replication  error  when  relaying  through  the  team,  as  well  as  enemy  interception 
(United  States  Army,  1987,  p.  1-1). 

Audio  communication  is  easy  to  understand,  straightforward,  and  is  situationally 
adaptive.  While  generally  used  in  situations  that  do  not  require  noise  discipline,  a  loud 
battlefield  environment  may  reduce  the  effectiveness  of  this  modality  (United  States 
Marine  Corps,  2002,  pp.  3-35).  Voice  recognition  software  coupled  with  a  hands-free 
radio,  like  the  Safariland  Group’s  Tactical  Throat  Microphone  Headset  as  shown  in 
Figure  2,  is  an  example  of  achieving  the  audio  interface.  While  voice  recognition 
software  is  not  ideal  for  everyday  use  such  as  typing  emails  or  sending  text  messages, 
limiting  the  number  of  commands  and  the  manner  in  which  the  commands  are  stated 
allows  for  customized  software,  tailored  to  the  man-machine  teaming  requirements. 
Moreover,  the  Marine  Corps  uses  the  NATO  phonetic  alphabet  to  enable  more  accurate 
audio  recognition  in  radio  communications;  this  type  of  simplification  can  also  improve 
man-machine  communication. 
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Figure  2.  Safariland  Group’s  Tactical  Throat  Microphone  Headset.  Source: 

Safariland  Group  (n.d.). 

Haptic  communication  is  the  least  commonly  used  method  of  communication  in 
pure  human-human  interaction  because  it  relies  on  the  sense  of  touch.  However,  by 
introducing  a  machine  to  the  fire  team,  this  method  becomes  a  more  viable  option  by 
using  devices  such  as  Schatzle  et  al.’s  ergonomic  vibrotactile  feedback  apparatus,  shown 
in  Figure  3.  This  device  overcomes  the  limitations  of  the  audio  and  visual  interfaces  as 
well  as  provide  machine  acknowledgement  of  receiving  various  human  commands 
(Schatzle  et  al.,  2010,  p.  675). 
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Figure  3.  Ergonomic  Vibrotactile  Feedback  Device.  Source: 

Schatzle  et  al.  (2010). 

With  the  addition  of  a  machine  into  the  fire  team,  electromagnetic 
communication,  such  as  personal  digital  assistant,  iPad,  smart  glasses,  or  modular  system 
as  shown  in  Figure  4,  becomes  a  fourth  interface  option  for  communication.  According  to 
Fong  et  al.,  this  type  of  interface  is  gaining  in  popularity  due  to  reduced  weight, 
portability,  and  touch-sensitive  displays  (Fong  et  al.,  2001,  p.  301).  As  early  as  2000, 
Perzanowski,  Adams,  Schultz,  and  Marsh  showed  the  validity  of  using  electromagnetic 
devices  “as  a  part  of  a  multi-modal  interface  for  interacting  with  an  autonomous  robot” 
(Perzanowski  et  al.,  2000,  p.  1). 


Figure  4.  Black  Diamond  Advanced  Technology's  Modular  Tactical  System 

(MTS).  Source:  Soldier  Systems  (n.d.). 


13 


Because  several  information  exchange  technologies  are  in  their  infancy,  the  effort 
to  identify  and  codify  the  MOEs  and  MOPs  is  a  vital  task  to  ensure  the  final  product 
satisfies  the  real-world  requirements. 

E.  MOEs  AND  MOPs 

As  the  UTACC  program  matures,  well  measured  MOEs  and  MOPs  will  be 
paramount  to  the  selection  process  communication  and  integration  of  the  Marine-machine 
team.  This  thesis  will  focus  directly  on  outlining  MOEs  and  MOPs  for  achieving  HMI 
through  reliable  and  redundant  communication. 

Linking  the  results  of  tactical  actions  to  the  overall  mission  objectives,  the  Joint 
Chiefs  of  Staff  J-7  defines  the  concept  of  Assessment  using  two  metrics:  MOPs  and 
MOEs  (USJCS  J-7,  2011,  p.  viii).  MOPs  “evaluate  task  performance”  or  “task 
accomplishment”  (USJCS  J-7,  2011,  p.  ix).  They  are  typically  “developed  and  assessed  at 
the  component  level  for  military  tasks  or  at  the  agency  or  organizational  level  for  non¬ 
military  tasks”  (USJCS  J-7,  2011,  p.  III-7).  Because  the  UTACC  program  is  still  in  the 
research  and  development  phase,  the  MOP  development  falls  under  the  non-military  task 
category  and  is  the  responsibility  of  MCWL,  in  coordination  with  the  NPS. 

MOEs  are  the  “criterion  used  to  assess  changes  in  system  behavior,  capability,  or 
operational  environment  that  is  tied  to  measuring  the  attainment  of  an  end  state, 
achievement  of  an  objective,  or  creation  of  an  effect”  (USJCS,  2017,  p.  GL-13).  They 
provide  an  “accurate  baseline  model”  for  determining  whether  the  organization’s  actions 
are  achieving  desired  effects  (USJCS  J-7,  2011,  p.  III-9).  Once  the  MOEs  are  established, 
the  sensor(s)  used  to  achieve  HMI  should  be  evaluated  against  the  model  proposed  in  this 
thesis. 

F.  CHAPTER  CONCLUSION 

This  literature  review  summarized  timely  and  relevant  information  primarily 
focused  with  the  ability  of  man  and  machine  to  communicate  and  the  most  effective  ways 
to  measure  the  communication’s  performance  and  effectiveness.  The  UTACC  program  is 
a  multifaceted  problem  set  attempting  to  harness  the  potential  of  the  man-machine  team. 
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While  this  is  a  new  and  exciting  field,  it  does  not  come  without  its  challenges.  As 
Johnson  states,  the  solutions  to  this  challenging  problem  set  are  viewed  through  the  OPD 
framework  (Johnson,  2014).  However,  current  doctrine  does  not  include  necessary  MOEs 
and  MOPs  for  the  man-machine  concept.  Additionally,  understanding  and  developing  the 
relationship  of  and  the  communication  between  man  and  machine  is  a  difficult  issue  that 
this  thesis  will  seek  to  address.  This  thesis  will  capitalize  on  the  aforementioned  work  to 
determine  the  most  effective  metrics  for  measuring  communication  and  integration  of 
Marines  and  autonomous  systems. 
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III.  RESEARCH  METHODOLOGY  AND  RELATED  FACTORS 


This  chapter  outlines  the  authors’  methodology  used  by  the  authors  in  building 
appropriate  HMI  MOEs  and  MOPs.  First,  the  basic  systems  engineering  process  is 
reviewed,  followed  by  explanations  of  UTACC  terms  and  assumptions  relevant  to  the 
authors’  research.  Next,  the  derivation  of  MOE/MOPs  is  explained,  beginning  with  an 
overview  of  high-level  doctrine  and  reducing  down  to  the  detailed  tasks  Marines  train  for 
as  outlined  in  the  Marine  Corps  Task  List.  Finally,  OPD  requirements  are  reviewed  for 
their  applicability  to  the  relevant  MOE/MOPs,  based  on  the  uniqueness  of  the  machine 
teammate. 

A.  BASIC  SYSTEMS  ENGINEERING  PROCESS 

As  originally  stated  by  the  program  sponsors  and  pioneered  by  Kirkpatrick  and 
Rushing  (2016),  development  of  the  MOPs  and  MOEs  for  this  thesis  and  the  UTACC 
program  used  the  basic  systems  engineering  processes,  as  shown  in  Figure  5,  and  the 
UTACC  concept  of  operations  (CONOPS)  thesis  (Kirkpatrick  &  Rushing,  2016).  Rice  et 
al.  originally  conducted  an  analysis  of  the  basic  systems  engineering  processes  as  defined 
in  the  Systems  Engineering  Management  textbook  (Blanchard,  2008)  which  Kirkpatrick 
and  Rushing  also  referenced  (Kirkpatrick  &  Rushing,  2016).  Since  that  time,  Blanchard 
published  the  5th  edition  to  the  textbook  and  the  authors  verified  that  the  systems 
engineering  process  referenced  in  previous  theses  remains  the  same  (Blanchard,  2016). 
Based  on  Rice  et  al.’s  (2015)  recommendations  and  previous  work  conducted  by 
Kirkpatrick  and  Rushing  (2016),  the  authors  viewed  “UTACC  as  a  system  of  systems 
(SoS)  capable  of  independent  operations  while  operating  within  the  Marine  Corps’ 
command  and  control  model  to  ensure  unity  of  effort  when  conducting  operations” 
(Kirkpatrick  &  Rushing,  2016,  p.  15).  According  to  their  findings,  “The  steps  that  were 
most  applicable  to  this  thesis  were:  definition  of  problem,  operational  requirements,  and 
functional  analysis.  The  entire  process  also  incorporated  feedback  mechanisms  as  an 
important  element  of  concept  generation”  (Rice  et  al.,  2015,  p.  21).  The  authors  focused 
their  research  on  these  three  steps  to  effectively  determine  quantifiable  metrics  for  the 


17 


man-machine  interface  as  specifically  viewed  through  the  lens  of  ensuring  that  humans 
and  machines  could  communicate  effectively  over  multiple  modalities. 


Identified 

need 


Figure  5.  Systems  Engineering  Model.  Source:  Blanchard  &  Blyler  (2016). 


Kirkpatrick  and  Rushing  (2016)  used  the  operational  requirements  identified  by 
the  UTACC  CONOPS  thesis  to  drive  the  development  of  system-level  MOPs  and  MOEs 
(Kirkpatrick  &  Rushing,  2016).  Those  measures  were  based  on  Performance  and  Related 
Operational  Parameters,  Utilization  Requirements,  and  Effectiveness  Requirements 
(Kirkpatrick  &  Rushing,  2016).  As  stated  by  Rice,  et  al.  (2015)  and  paraphrased  by 
Kirkpatrick  and  Rushing  (2016),  the  conduct  of  the  Functional  Analysis  served  as  the 
“heart  of  the  concept  generation”  for  the  UTACC  CONOPS  thesis  (Rice  et  al.,  2015,  p. 
23).  Kirkpatrick  and  Rushing  (2016)  used  the  results  to  develop  pertinent  MOPs  and 
MOEs.  The  authors  used  the  same  systems  engineering  process  to  develop  MOPs  and 
MOEs  for  the  man-machine  interface  resulting  in  key  metrics  for  future  testing  and 
implementation.  Follow  on  chapters  discuss  this  research  and  the  results  in  detail. 

B.  UTACC  TERMS  AND  DEFINITIONS 

Due  to  the  growing  number  of  theses  in  the  UTACC  program  and  the  inherent 
requirement  for  consistency  in  terms,  these  definitions  are  directly  sourced  from  Rice 
et  al.’s  work. 
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Small  tactical  unit-a  Marine  Corps  infantry  fire  team,  infantry  squad,  or 
reconnaissance  team. 

UTACC- armed  Marine(s)  conducting  operations  with  the  assistance  of  a 
mix  of  semi-autonomous  unmanned  ground  and  air  vehicles.  One  UTACC 
system  is  a  triad  of  a  human  component,  an  air  component,  and  a  ground 
component.  (SOW,  2016) 

Human  Component -  envisioned  as  a  small  tactical  unit  leader.  UTACC 
should  also  be  able  to  work  with,  provide  input  to,  and  receive  direction 
from  all  members  of  a  small  tactical  unit. 

User  Interface  System  (UIS)- a  combination  of  devices  that  stimulate 
multiple  senses  in  the  human.  For  example,  this  might  allow  him  to  do  the 
following:  see  a  map  of  the  operations  area  or  a  live  video  of  a  specific 
person  of  interest;  hear  a  warning  informing  him  that  a  component  has 
experienced  a  critical  system  failure;  or,  feel  a  warning  of  nearby  enemy 
force.  In  addition  to  providing  input  to  the  human,  the  UIS  will  also 
receive  input  from  the  human  and  then  relay  that  input  to  all  the  other 
UTACC  components.  The  human  inputs  can  also  come  in  a  variety  of 
ways:  hand  and  arm  signals  directing  the  tactical  movement  of  UTACC; 
verbal  messages  given  to  human  teammates  as  well  as  UTACC 
components;  touch  gestures/drawings  on  a  UTACC  generated  map  or 
preformatted  report. 

Air  Carrier  (AC)- an  unmanned  ground  vehicle  capable  of  carrying, 
launching,  recovering,  and  refueling  multiple  unmanned  air  vehicles 
(UAVs).  In  addition,  the  AC  will  be  capable  of  carrying  additional 
supplies  (e.g.,  ammunition,  food)  for  the  small  tactical  unit  as  well  as 
acting  as  a  communications  relay  for  the  UTACC  components.  In  the 
future,  this  vehicle  will  be  capable  of  high-speed  travel  over  rough  terrain 
and  off-road  areas. 

Unmanned  Air  Vehicle  (UAV)- an  aerial  platform  capable  of  carrying  any 
number  of  sensors  to  support  mission  specific  intelligence,  surveillance, 
and  reconnaissance  (ISR)  requirements  and  capable  of  vertical  takeoff  and 
landing.  The  UAV  will  be  capable  of  serving  as  a  vital  communications 
relay  node  between  geographically  separated  ground  components. 

Ground  Carrier  (GC)- an  unmanned  ground  vehicle  capable  of  carrying, 
deploying,  and  recovering  multiple  unmanned  ground  vehicles  (UGVs).  In 
addition,  the  GC  will  be  capable  of  carrying  additional  supplies  (e.g., 
ammunition,  food)  for  the  small  tactical  unit  as  well  as  acting  as  a 
communications  relay  for  the  UTACC  components.  This  vehicle  will  be 
capable  of  high-speed  travel  over  rough  terrain  and  off-road  areas. 
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Unmanned  Ground  Vehicle  (f/GUj-mission  specific  unmanned  systems 
capable  of  performing  discrete  ISR  missions.  The  UGVs,  similar  to  the 
UAVs,  could  have  a  variety  of  sensors  to  support  mission  specific  ISR 
requirements. 

Cue- is  a  notification  issued  by  the  UIS  to  the  Human  Component  where 
human  intervention  is  not  required. 

Alert- is  a  prompt  issued  by  the  UIS  to  the  Human  Component  requiring 
human  intervention.  (Rice  et  al.,  2015,  pp.  26-27) 

These  terms  remain  relevant  in  the  discussion  of  MOP/MOEs  since  they  relate  to 
the  components  of  the  UTACC  system. 

C.  UTACC  ASSUMPTIONS 

Due  to  the  lack  of  current  doctrine  and  research  in  the  field  of  man-machine 
teaming,  the  initial  UTACC  CONOPS  included  assumptions  that  were  necessary  to  frame 
the  problem;  these  assumptions  are  carried  forward  from  thesis  to  thesis  and  modified  as 
required.  Although  originally  assumed  that  UTACC  is  a  technology  agnostic  concept  by 
Rice  et  al.,  subsequent  analysis  of  alternatives  (AoA)  conducted  by  Roth  and  Buckler 
(2016)  narrowed  the  systems  and  technology  best  suited  for  current  UTACC 
developmental  testing  (Rice  et  al.,  2015,  p.  27).  However,  the  incorporation  of  newly 
developed  or  even  theoretical  technologies,  such  as  magnetic  field  communication, 
remains  vital  to  the  UTACC  program.  Just  as  Kirkpatrick  and  Rushing  developed  system 
specific  MOPs  and  MOEs  through  a  technologically  agnostic  methodology,  the  authors 
worked  to  do  the  same  for  the  man-machine  communication  interface  (Kirkpatrick  & 
Rushing,  2016). 

A  key  assumption  made  by  Kirkpatrick  and  Rushing  (2016),  which  the  authors 
carried  forward  into  this  thesis,  is  that  UTACC  could  apply  current  USMC  Task  List 
elements  to  verify  the  same  results  in  tasks  independent  of  whether  they  were  performed 
by  a  human  or  a  robot  (Kirkpatrick  &  Rushing,  2016).  Thus,  the  authors  assume  the 
results  of  the  actions  taken  during  information  exchange  between  the  man  and  machine 
team  will  be  commensurate  with  the  results  of  information  exchange  between  an  all 
human  fire  team,  although  the  modalities  or  interfaces  may  be  different. 
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Another  assumption  made  by  Kirkpatrick  and  Rushing  (2016)  and  still  valid  for 
this  thesis  is  that  the  MCTL  metrics  currently  used  in  the  UTACC  developmental  testing 
would  “accurately  reflect  metrics  applied  to  UTACC  in  future  testing”  (Kirkpatrick  & 
Rushing,  2016,  p.  18).  Manpower,  budget,  and  shifting  priorities  from  the  program 
sponsor  may  change  the  nature  of  future  UTACC  developmental  tests;  however,  those 
changes  should  not  affect  the  desired  end  state  of  the  UTACC  program. 

D.  UTACC  CONSTRAINTS 

The  proprietary  and  closed  nature  of  the  Intuitive  Robotic  Operator  Control 
(IROC)  event  held  in  October  2016  was  a  significant  constraint  in  developing  the  metrics 
for  man-machine  communication  was.  In  order  to  achieve  a  higher  level  of  participation 
from  industry  leaders,  MCWL  did  not  allow  outside  researchers  to  attend.  Although  the 
benefit  of  this  decision  is  a  more  open  forum  for  participants  of  the  event,  academic 
research  in  man-machine  teaming  suffered. 

Due  to  the  limited  nature  of  developmental  testing,  the  number  of  tasks  to  be 
evaluated  will  be  constrained.  As  a  result,  the  proposed  MOEs  and  MOPs  which  can  be 
evaluated  are  also  constrained.  Although  the  above  assumptions  state  that  MCTL  metrics 
will  be  used  in  the  testing,  it  is  currently  unknown  when  testing  will  be  able  to  accurately 
and  effectively  test  the  UTACC  system  fully,  so  MOEs  and  MOPs  must  be  designed  in  a 
modular  and  adaptable  way  as  the  system  evolves. 

E.  ROLE  OF  DOCTRINE  AND  TTPS 

As  originally  discussed  by  Kirkpatrick  and  Rushing  (2016),  Marine  Corps 
doctrine  establishes  the  fundamentals  for  operations  in  both  training  and  combat 
environments  through  the  publication  of  11  Marine  Corps  Doctrinal  Publications 
(MCDP).  “MCDPs  are  higher  order  doctrinal  publications  that  contain  the  fundamental 
and  enduring  beliefs  of  warfighting”  (Global  Security,  n.d.).  In  addition  to  the  MCDPs, 
Marine  Corps  Warfighting  Publications  (MCWP)  “have  a  narrower  focus  that  details 
tactics,  techniques  and  procedures  (TTPs)  used  in  the  prosecution  of  war  or  other 
assigned  tasks”  and  Marine  Corps  Reference  Publications  (MCRP)  “contain  general 
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reference  material  that  is  more  specific/detailed  than  the  MCWPs”  (Global 
Security,  n.d.). 

The  Marine  Corps  Task  List  (MCTL)  is  a  fourth  element  of  Marine  Corps 
doctrine  which,  “allows  for  quantifiable  measurement  of  proficiency  in  military  skills  and 
capabilities”  (Kirkpatrick  &  Rushing,  2016,  p.  19).  According  to  the  MCTL  Branch 
website: 

MCTL  is  the  authoritative,  standardized,  and  doctrinally-based  lexicon 
of  USMC  capabilities  defined  as  Marine  Corps  Tasks  (MCTs)  and 
used  by  units,  installations  and  the  supporting  establishments  in 
the  development  of  Mission  Essential  Tasks  and  Task  Lists 
(METs/METLs).  METs/METLs  are  the  list  of  “essential,”  critical, 
discrete,  externally-focused  MCTs  that  directly  enables  the  execution  of 
the  organizational  mission.  Capabilities,  defined  as  “MCTs”  and  resident 
in  MCTL  enable  Commanders  to  document  their  command  warfighting 
operational  abilities  as  METs/METLs,  providing  force  sourcing  planners, 
trainers  and  concept  developers  with  single  common  language  “tasks” 
articulating  both  Joint  and  USMC -specific,  manpower,  equipment  and 
training  requirements.  (United  States  Marine  Corps,  2016) 

Each  Marine  Corps  Task  (MCT)  has  a  collection  of  relevant  MOPs  and  MOEs  for 
timely,  quantifiable  feedback  pertaining  to  the  unit’s  ability  to  perform  the  stated  task  to  a 
given  standard.  Table  1  is  an  overview  of  how  a  MCT  is  defined,  broken  down  into  its 
subcomponents,  and  how  each  subcomponent  is  assigned  its  respective  metrics. 


Table  1.  Excerpt  from  MCTL  2.0.  Source:  United  States  Marine  Corps  (2016). 
MCT  5.3. 1.2  Exercise  Tactical  Command  and  Control 

Tactical  command  and  control  provides  purpose  and  direction  to  the  varied  activities  of  a 
military  unit.  It  is  the  means  by  which  the  Commander  recognizes  what  needs  to  be  done  and 
sees  to  it  that  appropriate  actions  are  taken.  Tasks  include:  to  order  warfare  degrees  of 
readiness;  to  direct  asset  assigmnent,  movement,  and  employment;  and,  to  control  tactical  assets, 
including  allied  and  joint  forces  assigned.  (JP  1-02,  3-0,  5-0,  5-00.2,  MCDP  1-0,  6,  NDP  6, 
NWP  3-21,  3-21.0  Rev  A,  3-56.1  Rev  A,  6-00.1,  NTA  5.4.1.2) 


Ml 

Time 

For  units  to  respond  to  tasking. 

M2 

Time 

Delay  in  response  to  orders. 

M3 

Percent 

Of  units  responding  appropriately  to  orders. 

M4 

Percent 

Of  mission  objectives  attained. 
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There  are  hundreds  of  MCTs  breaking  down  every  aspect  of  Marine  Corps 
operations  with  thousands  of  associated  MOP/MOEs.  Existing  MCTs  do  not,  however, 
account  for  the  evaluation  of  man-machine  teams  that  may  have  a  different  set  of  grading 
criteria  to  assess  their  capabilities.  Chapters  IV  and  V  discuss  the  criteria  further. 

F.  ANALYSIS  DEVELOPMENT  LAYERS 

To  capture  accurately  the  technical  performance  parameters  while  still 
incorporating  the  tactical  requirements,  the  developmental  layers  of  analysis  must  be 
clearly  understood.  These  layers  provide  the  framework  in  which  the  MOEs  and  MOPs 
are  nested  and  are  depicted  in  Figure  6.  The  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS) 
J-7  serves  as  the  30,000-feet  view  for  the  author’s  approach  by  articulating  the  joint 
definitions  and  purposes  of  MOEs  and  MOPs.  Next,  the  Director,  Operational  Test  and 
Evaluation  (DOT&E)  is  responsible  for  the  operational  testing  and  evaluation  of  major 
DOD  acquisition  programs.  They  provide  a  more  robust  approach  to  developing  metrics 
to  accurately  measure  a  system’s  effectiveness.  The  final  layer  is  the  proposed  UTACC 
MOEs  and  MOPs  that  are  further  refined  by  selected  MCTs  of  interest  and  the  OPD 
Interdependence  Analysis  (IA)  Tables  presented  in  Zach’s  Coactive  Design  thesis 
(Zach,  2016). 
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CJCS  J-7  Commander's  Handbook 
for  Assessment 


Figure  6.  Framework  for  HMI  MOE  and  MOP  Development 
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1. 


CJCS  J-7  Commander’s  Handbook  for  Assessment 


The  CJCS  J-7’s  Commander’ s  Handbook  for  Assessment  Planning  and  Execution 
provides  a  starting  point  for  the  development  of  the  UTACC  system  MOEs  and  MOPs. 
Linking  the  results  of  tactical  action  to  overall  mission  objectives,  MOEs  and  MOPs  are 
developed  metrics  used  to  assess  a  system: 

The  assessment  process  uses  MOPs  to  evaluate  task  performance  and 
MOEs  to  determine  progress  of  operations  toward  achieving  objectives, 
and  ultimately  the  end  state.  MOEs  help  answer  questions  like:  “are  we 
doing  the  right  things,  are  our  actions  producing  the  desired  effects,  or  are 
alternative  actions  required?”  MOPs  are  closely  associated  with  task 
accomplishment.  MOPs  help  answer  questions  like:  “was  the  action  taken, 
were  the  tasks  completed  to  standard,  or  how  much  effort  was  involved?” 

...  The  intent  in  developing  MOEs  and  their  associated  indicators  is  to 
build  an  accurate  baseline  model  for  determining  whether  joint  and 
supporting  agency  actions  are  driving  target  systems  toward  or  away  from 
exhibiting  the  desired  effects.  As  strategic  and  operational  level  effects  are 
seldom  attained  or  exhibited  instantaneously,  MOEs  provide  a  framework 
for  conducting  trend  analysis  of  system  behavior  or  capability  changes  that 
occur  over  time,  based  on  the  observation  of  specific,  discrete  indicators. 
(USJCS  J-7,  2011,  pp.  11-12) 

The  key  purpose  in  developing  MOEs  and  MOPs  is  to  drive  a  system  toward  mission 
success. 

2.  Director,  Operational  Test  and  Evaluation 

DOT&E  is  the  U.S.  government’s  primary  office  responsible  for  the  operational 
testing  and  evaluation  of  major  DOD  acquisition  programs.  While  the  UTACC  program 
is  currently  in  the  developmental  testing  phase,  it  is  important  the  guidelines  and 
procedures  laid  out  by  DOT&E  are  taken  into  consideration  early  in  the  design  process. 
As  depicted  in  the  “Vee”  Developmental  Model  in  Figure  7,  developer  and  user 
perspectives  are  incorporated  throughout  the  entire  systems  engineering  process.  These 
perspectives,  when  captured  early  in  the  design  process,  help  prevent  system  setbacks  and 
ensure  the  program  continues  to  meet  user  requirements. 
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Figure  7.  Generic  “Vee”  Developmental  Model.  Source: 

Blanchard  &  Blyler  (2016). 

Laid  out  in  their  “Mission  Focused  Evaluation  -  Guidance,”  DOT&E  articulates 
several  concepts  which  will  be  used  in  Chapters  IV  and  V  to  identify  and  formulate  the 
MOEs  and  MOPs  necessary  for  the  success  of  UT ACC’s  Marine-machine  interface 
solution  (DOT&E,  n.d.,  p.  1).  Foremost,  metrics  are  essential  for  the  success  of  any  test 
design  effort.  Selecting  the  right  metrics  requires  a  thorough  understanding  of  the  critical 
operational  issues  (COIs),  or  more  plainly  stated,  “The  key  operational  effectiveness  or 
suitability  issues  that  must  be  examined  in  operational  test  and  evaluation  to  determine 
the  system's  capability  to  perform  its  mission”  (DAU,  n.d.). 

As  DOT&E  highlights,  “The  metrics  will  provide  a  determination  of  mission 
capability,  lend  well  to  good  experimental  design  [Design  of  Experimentation],  and 
encapsulate  the  reasons  for  procuring  the  system”  (DOT&E,  n.d.,  p.  1).  When  creating 
metrics,  DOT&E  uses  two  types,  either  discrete  or  continuous.  Continuous  metrics 
incorporate  some  type  of  quantitative  feedback  into  their  outputs.  An  example  output  of  a 
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continuous  metric  is  shown  in  Table  2.  Although  the  “meets  threshold”  box  is  still  a  yes 
or  no,  the  key  to  the  metric  is  in  the  variable. 


Table  2.  Continuous  Metric  Example 


Task 

Variable 

Meets  Threshold 

Relay  information  to  fire  team  leader 

100  meters 

Y 

Conversely,  discrete  metrics  can  best  be  described  as  pass/fail  metrics.  An  example  of  a 
discrete  metric  is  shown  in  Table  3. 


Table  3.  Discrete  Metric  Example 


Task 

Meets  Threshold 

Relay  information  to  fire  team  leader 

Y 

Unlike  a  continuous  metric  that  provides  context  for  measuring  effectiveness,  a  discrete 
metric  discards  information  and  is  limited  in  its  usefulness.  In  the  discrete  metric 
example,  it  is  unknown  whether  the  machine  is  sitting  immediately  beside  the  fire  team 
leader  or  located  at  a  much  greater  distance;  we  lose  content.  For  the  purpose  of  this 
thesis,  discrete  metrics  are  used  to  formulate  MOEs.  Furthermore,  to  provide  context, 
continuous  metrics  are  nested  inside  the  discrete  metrics  as  MOPs.  This  idea  is  captured 
in  Table  4. 


Table  4.  Discrete/Continuous  Metric  Application  to  MOEs  and  MOPs 


MOE 

Task 

Threshold 

1.0 

Sensor  is  resilient  to  operating  environment 

Y 

MOP 

Task 

Variable 

Threshold 

1.0.1 

Sensor  is  waterproof 

50  m 

Y 

1.0.2 

Sensor  is  windproof 

40  kts 

Y 

1.0.3 

Sensor  is  temperature-proof 

-30° to  180° F 

Y 
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Lastly,  DOT&E  stresses  that  “metrics  chosen  must  also  be  well-defined  and 
meaningful.  Evaluators  should  consider  example  operational  scenarios  to  ensure  that  the 
metric  can  be  unambiguously  measured  (scored)  and  calculated  in  all  cases”  (DOT&E, 
n.d.,  p.  2).  Due  to  the  dynamic  operating  environment  of  the  Marine  Corps,  this  principle 
of  matching  the  metrics  to  environments  will  serve  as  a  fundamental  building  block  in  the 
following  chapters. 

3.  UTACC  MOEs  and  MOPs 

As  mentioned  at  the  beginning  of  this  section,  when  developing  MOEs  and 
MOPs,  the  unique  incorporation  of  a  machine  into  the  fire  team  leads  to  supplementary 
considerations.  In  addition  to  the  traditional  tactical  requirements,  a  fire  team  must  be 
able  to  meet  technical  tasks  which  must  also  be  included  to  effectively  evaluate  and 
measure  machine- specific  contributions. 

Based  on  the  scope  of  testing  during  MCWL’s  Developmental  Test  1  (DTI),  the 
tactical  requirements  are  limited  to  simple  fire  team  concepts  such  as  maintaining  current 
position  within  the  fire  team  and  changing  formations  when  given  the  proper  signal.  The 
preponderance  of  these  tactical  tasks  are  linked  to  MCTL  2.0  and  provide  the  starting 
point  for  refining  the  UTACC  MOEs  and  MOPs  into  HMI  specific  MOEs  and  MOPs. 
Table  5  depicts  the  preliminary  UTACC  MCTs  of  interest. 
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Table  5.  Preliminary  UTACC  MCTs  of  Interest.  Source:  United  States 

Marine  Corps  (2016). 


MCT 

Description 

5 

Exercise  Command  and  Control 

5.1 

Acquire,  Process,  Communicate  Information,  and  Maintain  Status 

5.1.3 

Maintain  Information  and  Force  Status 

5. 1.3.2 

Provide  Positive  ID  of  Friendly  Forces  Within  AO 

5.3 

Direct,  Fead,  Coordinate  Forces/Operations 

5.3.1 

Direct  Operations 

5.3. 1.2 

Exercise  Tactical  Command  and  Control 

Technical  tasks  are  much  more  difficult  to  capture,  as  there  is  no  real  starting 
point  to  draw  from  in  current  Marine  Corps  doctrine  or  TTPs.  The  authors  relied  on  the 
Coactive  Design  IA  tables  (Zach,  2016)  and  concurrent  UTACC  Immediate  Actions 
research  (Chenoweth  &  Wilcox,  2017)  to  form  the  initial  framework  to  begin  capturing 
required  technical  parameters.  As  seen  in  Table  6,  the  implied  tactical  tasks  are  in  the 
leftmost  column.  By  breaking  each  task  down  into  specific  OPD  requirements  necessary 
for  the  success  of  the  man-machine  team,  specific,  technically  focused  MOPs  are  derived. 
These  OPD  requirements  help  narrow  down  the  necessary  technical  measures  for 
incorporation  into  the  UTACC  MOEs  and  MOPs. 
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Table  6.  Coactive  Design  IA  Tables.  Source:  Zach  (2016). 


Maintain 

COP 


Tactical 

Alans 

and 

Cueing 


Send  Imagery 
and  Data  Back 
to  COP  and  to 
Leaders 


Provide  Alert 

Message  to 
Team  When 
Cntical 
Tactical 
Events  Occur 
(Team 
Response 
Required) 


Provide  Cues 
to  T  earn  When 
Less  Than 
Cntical 
Tactical 
Events  Occur 
(Team 
Response 
Optional) 


Recognize 
Tactical  Alet 
Scenario 


Recognize 

Cueing 

Scenano 


ley  may  be  positcned  dunng  this  ponton  of  the  missioi 
to  extend  the  communicaion  lines,  where  the  UxVs  serve 
as  intermediate  relay  nodes  inthecommuiicaJonSnlt 
between  the  objective  back  to  a  HHQ  or  adjacently 
operating  unit  that  would  othenwse  expenaicedegraded 
or  no  communication  links 


The  UxVs  should  always  notfy  the  team  of  cntical  tacica 
events,  including:  viien  in  the  vicinity  of  checkpoints  and 
other  important  gnds,  vinen  a  high  value  terget  or  be  on 
the  look  out  was  sposed,  directon  and  distanoe  of  enemy 
contact,  etc. 


The  T  earn  Leader  may  also  want  the  UxVs  to  noify  him  of 
additional  events  like  approaching  traffc ,  or  potenial  hot 
spots  along  the  route  Wiere  pos9ble  EDs  may  be 
emplaced,  etc 


G.  CHAPTER  CONCLUSION 

The  development  of  MOEs  and  MOPs  is  a  difficult  task,  which  requires  input  and 
validation  from  multiple  sources.  Add  in  the  complexities  of  a  man-machine  team  and  the 
number  of  issues  grows  exponentially.  However,  by  using  the  methodology  in  Table  6, 
the  authors  were  able  to  narrow  their  focus  to  the  design  of  those  specific  MOEs  and 
MOPs  necessary  for  the  successful  communication  between  man  and  machine. 


30 


IV.  UTACC  HMI  MEASURES  OF  EFFECTIVENESS  AND 
MEASURES  OF  PERFORMANCE 


A  preliminary  look  at  the  MCT  5  family  of  tasks,  “Exercising  Command  and 
Control,”  served  as  the  basis  for  the  creation  of  UTACC  HMI  MOEs  and  MOPs.  From 
the  Marine  Expeditionary  Force  (MEF)  level  to  the  fire  team  level,  Marine  Corps 
Command  and  Control  is  an  absolute  necessity  in  ensuring  the  success  of  Marine  Corps 
operations.  Due  to  the  uniqueness  and  complexities  accompanying  the  UTACC  program, 
additional  MOEs  and  MOPs  were  derived  from  insight  provided  by  the  prior  research  of 
Zach  (2016),  Johnson  (2014),  and  Rice,  et  al.  (2015)  as  well  as  the  current  research  of 
Chenoweth  and  Wilcox  (2017).  While  the  current  MCTL  lacks  metrics  for  the  UTACC 
program,  nesting  newly  developed  sub-tasks  within  the  current  framework  allows  for  a 
rapid  integration  with  minimal  disruption. 

A.  MOPs 

As  stated  in  Chapter  II,  MOPs  evaluate  task  performance  or  task  accomplishment 
(USJCS  J-7,  2011,  p.  ix).  With  this  in  mind,  the  authors  worked  with  the  assumption  that, 
as  long  as  a  metric  is  applied,  the  task’s  performance  can  be  assessed.  The  MOPs  are  not 
designed  to  state  whether  the  mission  was  accomplished,  but  only  to  show  how  the 
human  and  machine  are  interacting  compared  to  the  objective  standards. 

As  an  example,  a  metric  for  audio  communication  from  the  machine  to  the  human 
may  be  “did  the  human  hear  the  machine.”  If  the  answer  is  “yes,”  then  the  metric  was 
achieved.  However,  the  MOP  does  not  explain  whether  the  machine  communicated  the 
correct  information  or  if  the  human  understood  the  correct  information.  It  only  answers 
the  question  of  “did  the  human  hear  the  machine.”  Both  a  metric  and  a  way  to  take  the 
measurement  are  vital  to  succeed  in  effectively  measuring  the  MOPs;  if  either  is  lacking, 
evaluation  of  the  MOP  may  not  be  correct.  Also  critical  to  the  MOPs  and  their  use  is  the 
threshold  that  must  be  achieved.  If,  for  example,  the  MOP  measures  a  percentage  of 
correct  commands  communicated,  then  that  result  must  be  compared  against  a  threshold 
to  determine  whether  it  meets  an  acceptable  value. 
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Table  7  depicts  an  excerpt  from  MCTL  2.0  and  the  MOPs  associated  with  MCT 
5.3. 1.2-  Exercise  Tactical  Command  and  Control. 


Table  7.  Excerpt  from  MCTL  2.0.  Source:  United  States  Marine  Corps  (2016). 


Ml 

Time 

For  units  to  respond  to  tasking. 

M2 

Time 

Delay  in  response  to  orders. 

M3 

Percent 

Of  units  responding  appropriately  to  orders. 

M4 

Percent 

Of  mission  objectives  attained. 

B.  MOEs 

As  stated  in  Chapter  II,  MOEs  are  the  “criterion  used  to  assess  changes  in  system 
behavior,  capability,  or  operational  environment  that  is  tied  to  measuring  the  attainment 
of  an  end  state,  achievement  of  an  objective,  or  creation  of  an  effect”  (USJCS,  2017,  p. 
GL-13).  The  authors  of  this  thesis  focused  on  the  MOEs’  use  in  assessing  whether  or  not 
the  objective  was  achieved  during  the  testing.  The  MOE  associated  with  Table  7’s  MOPs 
is  MCT  5.3. 1.2,  Exercise  Tactical  Command  and  Control,  or  more  simply,  “Was  tactical 
command  and  control  exercised  effectively?”  An  MOE  is  defined  in  such  a  way  that  the 
associated  MOPs  support  the  MOE.  Therefore,  if  the  threshold  standards  for  the 
associated  MOPS  are  met,  then  the  MOE  will  be  met. 

C.  MCTL  ORGANIZATION  AND  WARFIGHTING 

After  a  thorough  review  of  MCTL  2.0,  the  authors  focused  on  the  addition  of  a 
MCT  sub-task  rather  than  rewriting  the  current  MCTL  due  to  the  limited  nature  of 
robotics  throughout  the  Operating  Forces.  Based  on  the  scope  of  this  thesis,  the  authors 
recommend  that  an  additional  sub-task  be  listed  as  5.1.4-  Maintain  Two-Way 
Communications  with  Autonomous  Robotics  which  would  be  subordinate  to  5.1-  Acquire, 
Process,  Communicate  Information,  and  Maintain  Status  and  ultimately  subordinate  to 
MCT  5-  Exercise  Command  and  Control  (United  States  Marine  Corps,  2016).  Table  8 
depicts  this  recommended  hierarchy. 
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Table  8.  Recommended  Update  to  MCTL  2.0.  Adapted  from  United  States 

Marine  Corps  (n.d.). 


MCT 

Title 

5 

Exercise  Command  and  Control 

5.1 

Acquire,  Process,  Communicate  Information,  and  Maintain  Status 

5.1.1 

Provide  and  Maintain  Communications 

5.1.2 

Manage  Means  of  Communicating  Information 

5.1.3 

Maintain  Information  and  Force  Status 

5.1.4 

Maintain  Two-Way  Communication  with  Autonomous  Robotics 

The  recommended  location  of  this  new  sub-task  was  chosen  due  to  the  intrinsic 
communication  requirement  of  UTACC  “acting  collaboratively  with  each  other  and  with 
humans”  (NPS  &  MCWL,  2016,  p.  1).  The  chosen  interfaces  for  the  communication 
between  UTACC  and  Marines  must  not  overburden  the  Marine’s  cognitive  load  while 
maintaining  effective  command  and  control.  Table  9  depicts  the  sub-tasks  subordinate  to 
MCT  5.1.4  which  will  measure  the  performance  over  the  different  communication 
modalities  as  discussed  in  Chapter  II. 


Table  9.  Recommended  Update  to  MCT  5.1.4 


MCT 

Title 

5.1.4 

Maintain  Two-Way  Communication  with  Autonomous  Robotics 

5. 1.4.1 

Identification  of  Team  Members 

5. 1.4.2 

Explicit  Human-Initiated  Communication 

5. 1.4.3 

Explicit  Robot-Initiated  Communication 

D.  TESTING  ENVIRONMENT 

The  key  to  an  accurate  evaluation  of  the  UTACC  program  through  the  authors’ 
MOEs  and  MOPs  is  ensuring  the  correct  item  is  evaluated  at  the  correct  time  in  the 
correct  way.  More  simply,  UTACC  should  not  be  penalized  because  another  fire  team 
member  was  not  on  task.  The  team  failing  in  this  manner  would  be  the  fault  of  a  human 
fire  team  member,  not  the  robot. 
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During  UTACC’s  DTI,  this  situation  manifested  itself  several  times.  DTI  was  a 
preliminary  test  that  focused  on  the  UTACC  software.  To  assist  in  evaluating  and  logging 
data  during  the  test,  software  toolkits  were  utilized  which,  among  other  items  of  interest, 
displayed  the  global  positioning  system  (GPS)  data  of  the  human  fire  team  members  and 
the  robot.  One  of  the  issues  encountered  throughout  testing  was  GPS  error  in  locating 
both  the  humans  and  the  robot.  GPS  data  was  continually  transmitted  to  the  UTACC 
software,  which  then  computed  a  velocity  vector.  The  robot  used  the  velocity  vector  to 
maintain  its  position  or  move  into  the  correct  position.  When  inaccurate  GPS  data  was 
directed  to  UTACC,  an  invalid  velocity  vector  was  sent  to  the  robot.  Consequently, 
UTACC’s  algorithms  would  maneuver  the  robot  out  of  position.  At  a  surface  level,  one 
could  argue  that  the  software’s  algorithms  failed  in  that  they  did  not  compute  velocity 
vectors  that  accurately  positioned  the  robot  due  to  poor  filtering  of  the  data.  However, 
after  closer  examination  of  the  testing  toolkits,  the  UTACC  algorithms  were  accurately 
computing  velocity  vectors  based  off  the  GPS  data  received.  In  other  words,  the  UTACC 
software  was  correctly  doing  what  it  was  supposed  to  do  while  something  outside  the 
testing — the  navigation  data — was  failing. 

With  this  example  in  mind,  future  testing  of  UTACC  HMIs  must  ensure  the 
testing  environment  creates  a  scenario  that  accurately  captures  the  right  data  points  for 
the  MOPs  under  test.  With  four  different  modes  of  possible  man-machine 
communication,  isolating  and  independently  testing  each  mode  is  critical  in  evaluating 
the  UTACC  HMI  system  as  a  whole.  Current  thesis  work  by  Beierl  and  Tschirley  (2017) 
is  exploring  UTACC  situational  awareness  and  seeks  to  provide  insight  on  how  to 
accurately  test  the  fire  team  members’  situational  awareness  throughout  execution  of  a 
mission.  Their  work  will  provide  the  framework  for  future  testing  of  UTACC’s  HMI 
system. 

E.  CHAPTER  CONCLUSION 

Instituting  change  in  any  bureaucratic  organization,  let  alone  the  Marine  Corps,  is 
a  daunting  task.  For  change  to  be  successful,  minimizing  disturbance  to  the  organization 
while  easing  the  transition  helps  facilitate  the  integration  of  new  ideas.  This  thought 
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process  guided  the  authors  in  their  decision  to  nest  the  UTACC  HMI  MOEs  and  MOPs 
within  current  MCTL  2.0  tasks.  Furthermore,  the  decision  was  made  to  work  exclusively 
within  the  proposed  MCT  5.1.4-  Maintain  Two-Way  Communications  with  Autonomous 
Robotics  task  and  not  modify  or  add  new  MOEs  and  MOPs  to  currently  published  tasks. 
Lastly,  defining  an  accurate  testing  environment  for  the  evaluation  of  UTACC’ s  HMI 
system  is  a  crucial  step  in  effectively  applying  and  validating  the  MOEs  and  MOPs 
proposed  in  the  final  tables,  and  are  discussed  in  Chapter  V. 
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V.  SUMMARIZING  RESULTS  AND  RECOMMENDATIONS  FOR 

FURTHER  RESEARCH 


This  chapter  provides  a  summary  of  the  authors’  research  results,  broken  down  by 
the  type  of  communication  requiring  evaluation.  Additionally,  limitations  of  the  resulting 
tables  are  discussed  which,  the  authors  believe,  will  require  further  research  and 
development  as  the  UTACC  program  progresses.  Lastly,  future  research  topics  are 
recommended  to  drive  UTACC  closer  to  integration  in  the  Marine  Corps  writ  large. 

A.  SUMMARIZING  RESULTS 

1.  MOP  and  MOE  Final  Tables 

As  discussed  in  previous  chapters,  the  MOEs  and  MOPs  required  to  evaluate  the 
effectiveness  of  the  HMI  communication  were  broken  down  into  sub-tasks  subordinate  to 
the  recommended  MCT  5.1.4.  The  next  sections  will  discuss  each  of  these  sub-tasks 
(originally  shown  in  Table  9)  in  more  depth. 

The  following  tables  will  focus  on  the  identification  of  team  members  by  UTACC 
and  communication  that  is  human-UTACC  and  UTACC-human.  The  authors  did  not 
recommend  either  human-human  or  UTACC-UTACC  MOEs/MOPs  as  it  was  outside  of 
the  scope  of  this  thesis. 

a.  MCT  5. 1.4.1 — Identification  of  Team  Members 

Table  10  depicts  the  MOPs  associated  with  MCT  5. 1.4.1.  In  order  to  evaluate  the 
MOPs  listed,  the  authors  created  additional  sub-tasks  with  their  own  MOPs  as  depicted  in 
Tables  11,  12,  and  13.  By  assessing  MCTs  5. 1.4. 1.1,  5. 1.4.1. 2,  and  5. 1.4. 1.3,  MCT 
5. 1.4.1  will  be  evaluated  as  a  whole. 
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Table  10.  MCT  5 . 1 .4 — Identification  of  Team  Members 


5. 1.4.1 

Metric 

Identification  of  Team  Members 

Ml 

Percent 

Of  time  UTACC  can  identify  fire  team  members 

M2 

Percent 

Of  modalities  UTACC  can  successfully  use  to  identify  fire  team  members 

M3 

Distance 

Between  team  members  during  successful  identification 

Table  11 

.  MCT  5. 1 .4. 1 . 1 — Visual  Identification  of  Team  Members 

5. 1.4. 1.1  Metric 

Visual  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  visually 

M2 

Percent 

Of  time  UTACC  can  visually  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  visually  identify  the  primary  human  (fire  team  leader) 

Table  12.  MCT  5. 1.4. 1 .2 — Audible  Identification  of  Team  Members 


5. 1.4. 1.2 

Metric 

Audible  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  audibly 

M2 

Percent 

Of  time  UTACC  can  audibly  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  audibly  identify  the  primary  human  (fire  team  leader) 

Table  13.  MCT  5. 1.4.1. 3 — Electromagnetic  Identification  of  Team  Members 


5. 1.4.1. 3 

Metric 

Electromagnetic  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  electromagnetically 

M2 

Percent 

Of  time  UTACC  can  electromagnetically  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  electromagnetically  identify  the  primary  human  (fire  team  leader) 
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In  order  to  measure  the  effectiveness  of  UTACC  in  identifying  the  team  members, 
and  specifically  identifying  the  team  leader,  the  authors  separated  the  task  of 
identification  into  the  modalities  that  were  discussed  in  Chapter  II.  The  key  factors 
measured  are  the  ability  for  UTACC  to  identify  the  team  leader,  the  time  it  takes  UTACC 
to  identify  the  team  leader,  and  the  ability  for  UTACC  to  identify  other  members  of  the 
team. 


b.  MCT  5. 1.4.2 — Explicit  Human-Robot  Communication 

MCT  5. 1.4. 2  pertains  to  all  communication  originating  from  the  human  and 
directed  to  UTACC.  Tables  14-18  depict  the  MCTs  along  with  their  respective  MOPs 
associated  with  evaluating  the  effectiveness  of  explicit  human-UTACC  communication. 

In  order  to  measure  the  effectiveness  of  the  communication,  the  authors  ensured 
that  the  following  questions  would  be  answered  through  evaluation: 

•  Did  UTACC  receive  the  message? 

•  Did  UTACC  relay  the  message? 

•  Did  the  human  verify  UTACC ’s  relay? 

Through  answering  the  above  questions  in  each  of  the  modalities,  the  authors 
believe  that  the  effectiveness  of  human-UTACC  communication  can  be  effectively 
measured  and  evaluated. 


Table  14.  MCT  5. 1.4. 2 — Explicit  Human-Initiated  Communication 


5. 1.4.2 

Metric 

Explicit  Human-Initiated  Communication 

Ml 

Percent 

Of  successful  human-initiated  messages 

M2 

Percent 

Of  successful  human-initiated  messages  on  first  transmission 

M3 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages 

M4 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages  on  first  transmission 

39 


Table  15.  MCT  5. 1.4.2. 1 — Visual  Human-Initiated  Communication 


5. 1.4. 2.1 

Metric 

Visual  Human-Initiated  Communication 

Ml 

Percent 

Of  visual  messages  UTACC  received 

M2 

Percent 

Of  visual  messages  UTACC  received  correctly 

M3 

Percent 

Of  visual  messages  UTACC  received  correctly  on  first  transmission 

M4 

Percent 

Of  visual  messages  UTACC  relayed  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

Table  16.  MCT  5. 1.4. 2. 2 — Audible  Human-Initiated  Communication 

5.1.4.2.2 

Metric 

Audible  Human-Initiated  Communication 

Ml 

Percent 

Of  audible  messages  UTACC  received 

M2 

Percent 

Of  audible  messages  UTACC  received  correctly 

M3 

Percent 

Of  audible  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent 

Of  audible  messages  UTACC  relays  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

Table  17.  MCT  5. 1.4. 2. 3 — Electromagnetic  Human-Initiated  Communication 

5. 1.4.2. 3 

Metric  Electromagnetic  Human-Initiated  Communication 

Ml 

Percent  Of  electromagnetic  messages  UTACC  received 

M2 

Percent  Of  electromagnetic  messages  UTACC  received  correctly 

M3 

Percent  Of  electromagnetic  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent  Of  electromagnetic  messages  UTACC  relays  correctly 

M5 

Time  For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent  Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance  Between  human  and  UTACC  during  messaging 

Table  18.  MCT  5. 1.4. 2.4 — Haptic  Human-Initiated  Communication 


5. 1.4. 2.4 

Metric 

Haptic  Human-Initiated  Communication 

Ml 

Percent 

Of  haptic  messages  UTACC  received 

M2 

Percent 

Of  haptic  messages  UTACC  received  correctly 

M3 

Percent 

Of  haptic  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent 

Of  haptic  messages  UTACC  relays  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 
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c.  MCT  5. 1.4.3 — Explicit  Robot-Human  Communication 

Similar  to  how  MCT  5. 1.4. 2  pertains  to  human-initiated  communication,  MCT 
5. 1.4. 3  pertains  to  communication  originated  by  UTACC.  Tables  19-23  show  the 
corresponding  MCTs  and  their  MOPs  which  can  be  used  to  evaluate  the  effectiveness  of 
the  UTACC-initiated  communications. 

Just  as  in  the  previous  section,  three  questions  must  be  answered  to  verify 
effective  communication: 

•  Did  the  human  receive  the  message? 

•  Did  the  human  acknowledge  the  message? 

•  Did  UTACC  understand  the  acknowledgement? 

The  difference  between  these  and  the  previous  sections’  questions  pertain  to 
UTACC  relaying  as  opposed  to  the  human  acknowledging  the  original  message.  The 
distinction  between  these  two  ideas  is  that  UTACC,  just  like  a  human  member  of  the  fire 
team,  needs  to  relay  the  message  to  other  team  members;  those  members  will  then 
continue  the  relay  process  until  the  entire  team  is  notified.  Although  the  human  will  also 
be  relaying  the  message  to  other  team  members,  the  MOP  for  testing  UTACC  is  how  well 
it  can  track  the  human’s  acknowledgement,  not  the  ability  of  the  human  to  conduct  the 
relay.  The  human  relaying  the  message  to  another  human  is  beyond  the  scope  of  this 
thesis. 


Table  19.  MCT  5. 1.4. 3 — Explicit  Robot-Initiated  Communication 


5. 1.4.3 

Metric 

Explicit  Robot-Initiated  Communication 

Ml 

Percent 

Of  successful  robot-initiated  messages 

M2 

Percent 

Of  successful  robot-initiated  messages  on  first  transmission 

M3 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages 

M4 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages  on  first  transmission 
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Table  20.  MCT  5. 1.4.3. 1 — Visual  Robot-Initiated  Communication 


5. 1.4. 3.1 

Metric 

Visual  Robot-Initiated  Communication 

Ml 

Percent 

Of  visual  messages  human  received 

M2 

Percent 

Of  visual  messages  human  received  correctly 

M3 

Percent 

Of  visual  messages  human  received  correctly  on  first  transmission 

M4 

Percent 

Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent 

Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time 

For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

Table  21.  MCT  5. 1.4. 3. 2 — Audible  Robot-Initiated  Communication 

5.1.4.3.2 

Metric 

Audible  Robot-Initiated  Communication 

Ml 

Percent 

Of  audible  messages  human  received 

M2 

Percent 

Of  audible  messages  human  received  correctly 

M3 

Percent 

Of  audible  messages  human  received  correctly  on  first  transmission 

M4 

Percent 

Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent 

Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time 

For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

Table  22.  MCT  5. 1.4. 3. 3 — Electromagnetic  Robot-Initiated  Communication 


5. 1.4.3. 3  Metric 

Electromagnetic  Robot-Initiated  Communication 

Ml 

Percent 

Of  electromagnetic  messages  human  received 

M2 

Percent 

Of  electromagnetic  messages  human  received  correctly 

M3 

Percent 

Of  electromagnetic  messages  human  received  correctly  on  first  transmission 

M4 

Percent 

Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent 

Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time 

For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

Table  23.  MCT  5. 1.4. 3.4 — Haptic  Robot-Initiated  Communication 


5. 1.4. 3.4 

Metric 

Haptic  Robot-Initiated  Communication 

Ml 

Percent 

Of  haptic  messages  human  cognitively  received 

M2 

Percent 

Of  haptic  messages  human  cognitively  received  on  first  transmission 

M3 

Percent 

Of  human  acknowledgements  understood  by  UTACC 

M4 

Percent 

Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M5 

Time 

For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M6 

Distance 

Between  human  and  UTACC  during  messaging 
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2. 


Limitations  of  MOP  and  MOE  Tables 


The  authors  identified  three  main  limitations  while  developing  these  MOEs  and 
MOPs.  First,  due  to  changing  environment  and  conditions,  the  thresholds  for  testing  will 
not  be  static.  Given  a  testing  scenario  and  in  line  with  the  proposed  MCTs,  a  range  of 
thresholds  will  need  to  be  developed  to  realistically  assess  the  effectiveness  and 
performance  of  the  communication  modalities.  Second,  the  electromagnetic  modality  is  a 
rapidly  evolving  method  of  communication.  As  a  result,  rather  than  drive  the 
requirements  process,  it  was  left  intentionally  vague  to  serve  as  a  guideline  for  future 
development,  testing,  and  implementation. 

Third,  operating  UTACC  often  requires  rapid  tactical  adaptation.  From  an  HMI 
perspective,  this  rapid  adaptation  means  the  modality  used  for  communication,  and  thus 
the  interface  used,  changes  during  a  mission.  Thus,  an  MOP  might  show  that  a  given  set 
of  devices  work,  but  the  team  leader,  for  tactical  reasons,  may  not  choose  that  method. 
Therefore,  determining  the  overall  MOP  of  communications  between  Marines  and 
machines  is  made  more  complicated  due  to  the  dynamic  nature  of  tactical  operations. 
Thus,  several  modalities  may  need  to  be  used  during  a  mission.  Selection  of  the  correct 
modality  for  communication  based  on  the  tactical  situation  is  an  area  ripe  for  further 
research. 

B.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

a.  MOE/MOP  Tables  for  Situational  Awareness  Sensor  Suite 

During  DTI,  UTACC  relied  on  GPS  data  from  fire  team  members  to  form  its 
situational  awareness  with  respect  to  its  position  in  the  fire  team.  While  GPS  was  chosen 
to  facilitate  timely  testing,  issues  brought  on  by  localization  error  highlighted  a  critical 
area  requiring  future  research.  This  thesis  developed  MOE/MOP  tables  for  the  sensors 
used  in  fire  team  communications  with  UTACC.  Concurrent  to  this  thesis  is  research  by 
Beierl  and  Tschirley  (2017)  on  UTACC  situational  awareness.  However,  a  fundamental 
area  has  yet  to  be  researched:  MOEs  and  MOPS  used  to  assess  situational  awareness  of 
UTACC  and  the  fire  team  members.  This  thesis,  in  conjunction  with  Beierl  and 
Tschirley’s  (2017),  provides  the  framework  for  the  next  logical  step  in  Marine-machine 
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teaming:  developing  MOE/MOP  tables  to  assess  UTACC’s  situational  awareness  sensor 
suite,  as  well  as  developing  MOE/MOP  tables  to  assess  the  cognitive  load  on  fire  team 
member’s  situational  awareness  brought  on  by  the  incorporation  of  new  sensors. 

b.  MOE/MOP  Tables  for  Sensor  Suite  Supporting  Targeting  Data 

As  mentioned  by  Kirkpatrick  and  Rushing  (2016),  during  LTA-2,  networked 
UTACC  sensors  were  used  semi-autonomously  to  generate  targeting  data  which  was  then 
relayed  to  a  notional  strike  platform  for  follow-on  execution  of  the  strike  package 
(Kirkpatrick  &  Rushing,  2016).  They  go  on  to  recommend  research  that  explores 
UTACC’s  role  in  supporting  air- to- surface  targeting.  In  conjunction  with  this  research, 
establishing  MOEs  and  MOPs  for  the  target  acquisition  sensor  suite  needs  to  be 
researched  concurrently.  MCRP  3- 16.1. A  Tactics,  Techniques,  And  Procedures  for  Field 
Artillery  Target  Acquisition  serves  as  starting  point  for  both  of  these  areas. 

c.  Changing  the  Role/Scale  of  UTACC 

This  thesis  explored  and  created  MOEs  and  MOPs  to  assist  in  the  selection  of 
sensors  used  for  human-machine  communication  at  the  fire  team  level.  Furthermore,  the 
authors  constrained  themselves  to  UTACC  operating  in  the  role  of  automatic  rifleman  in 
order  to  remain  consistent  with  concurrent  research.  Because  the  authors  limited  the 
relationship  to  the  automatic  rifleman  within  a  fire  team,  MOEs  and  MOPs  focused 
specifically  on  that  relationship.  However,  as  UTACC  progresses,  MOE/MOP  tables  will 
need  to  be  established  to  help  developers  evaluate  relationships  such  as  robot-robot 
communication,  robot-higher  (squad,  company,  etc.)  communication,  or  UTACC 
communicating  directly  with  a  node  or  database  informing  the  common  operating  picture. 

d.  Doctrinal  Changes  with  the  Addition  of  UTACC 

As  mentioned  in  Chapter  II,  this  thesis,  paired  with  Kirkpatrick  and  Rushing’s 
(2016)  work,  will  aid  in  the  creation  of  new  doctrine  inclusive  of  the  autonomous 
systems  in  line  with  both  EF21  and  the  Marine  Corps  Operating  Concept.  Current 
doctrine  establishes  TTPs  based  on  the  firepower  provided  by  the  automatic  rifleman. 
However,  the  technological  advantages  afforded  a  Marine-machine  fire  team  will 
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certainly  require  an  evolution  of  current  fire  team-level  TTPs  to  maximize  UTACC’s 
potential.  Exploring  and  researching  what  these  doctrinal  changes  will  look  like  is 
another  step  forward  in  developing  the  UTACC  program. 

C.  CHAPTER  CONCLUSION 

MCWL’s  UTACC  program  seeks  to  address  the  challenges  encountered  in  the 
complex  battlefield  environment  of  the  twenty-first  century  with  the  integration  of 
Marines  and  autonomous  machines  at  the  fire  team  level.  This  thesis  explored  a  small 
piece  of  that  equation:  communication  interface  requirements  between  man  and  machine. 
While  the  communication  interfaces  are  a  fraction  of  the  overall  UTACC  problem  set, 
choosing  the  right  interfaces  to  achieve  man-machine  communication  is  essential  for  the 
continued  success  of  the  Marine-machine  teaming  concept.  The  authors  of  this  thesis 
established  recommended  MOEs/MOPs  for  evaluating  the  communication  interfaces 
required  for  Marine-machine  communication  as  well  as  where  they  fit  within  the  MCTL. 
Furthermore,  recommended  future  research  topics  will  continue  to  drive  the  progress  of 
the  UTACC  program.  While  still  in  the  early  stages  of  development,  the  UTACC 
program  has  the  potential  to  fundamentally  shift  the  way  mankind  approaches  warfare  in 
the  years  to  come. 


45 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


46 


APPENDIX  A.  MCT  5.1.4  MAINTAIN  TWO-WAY  COMMUNICATION 
WITH  AUTONOMOUS  ROBOTICS  AND  FIRST  LEVEL  SUB-TASKS 


This  shows  the  highest  level  breakdown  of  MCT  5.1.4. 


5.1.4  Metric  Maintain  Two-Way  Communication  with  Autonomous  Robotics 

Ml  Percent  Of  successfully  communicated  messages 

M2  Percent  Of  successfully  communicated  messages  on  first  transmission 

M3  Distance  Between  team  members  during  successful  communication 

5. 1.4.1  Metric  Identification  of  Team  Members 

Ml  Percent  Of  time  UTACC  can  identify  fire  team  members 

M2  Percent  Of  modalities  UTACC  can  successfully  use  to  identify  fire  team  members 

M3  Distance  Between  team  members  during  successful  identification 

5. 1.4. 2  Metric  Explicit  Human-Initiated  Communication 

Ml  Percent  Of  successful  human-initiated  messages 

M2  Percent  Of  successful  human-initiated  messages  on  first  transmission 

M3  Percent  Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages 

M4  Percent  Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages  on  first  transmission 

5. 1.4.3  Metric  Explicit  Robot-Initiated  Communication 

Ml  Percent  Of  successful  robot-initiated  messages 

M2  Percent  Of  successful  robot-initiated  messages  on  first  transmission 

M3  Percent  Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages 

M4  Percent  Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages  on  first  transmission 
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APPENDIX  B.  MCT  5.1.4.1  IDENTIFICATION  OF  TEAM  MEMBERS 

AND  SUB-TASKS 


This  shows  the  highest  level  breakdown  of  MCT  5. 1.4.1. 


5. 1.4.1 

Metric 

Identification  of  Team  Members 

Ml 

Percent 

Of  time  UTACC  can  identify  fire  team  members 

M2 

Percent 

Of  modalities  UTACC  can  successfully  use  to  identify  fire  team  members 

M3 

Distance 

Between  team  members  during  successful  identification 

5.1. 4.1.1 

Metric 

Visual  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  visually 

M2 

Percent 

Of  time  UTACC  can  visually  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  visually  identify  the  primary  human  (fire  team  leader) 

5. 1.4.1. 2 

Metric 

Audible  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  audibly 

M2 

Percent 

Of  time  UTACC  can  audibly  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  audibly  identify  the  primary  human  (fire  team  leader) 

5. 1.4.1. 3 

Metric 

Electromagnetic  Identification  of  Team  Members 

Ml 

Percent 

Of  fire  team  members  UTACC  can  identify  electromagnetically 

M2 

Percent 

Of  time  UTACC  can  electromagnetically  identify  the  primary  human  (fire  team  leader) 

M3 

Time 

To  electromagnetically  identify  the  primary  human  (fire  team  leader) 
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APPENDIX  C.  MCT  5.1.4.2  EXPLICIT  HU  MAN -ROBOT 
COMMUNICATION  AND  SUB-TASKS 


This  shows  the  highest  level  breakdown  of  MCT  5. 1.4.2. 


5. 1.4. 2 

Metric 

Explicit  Human-Initiated  Communication 

Ml 

Percent 

Of  successful  human-initiated  messages 

M2 

Percent 

Of  successful  human-initiated  messages  on  first  transmission 

M3 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages 

M4 

Percent 

Of  modalities  able  to  be  used  in  acheiving  successful  human-initiated  messages  on  first  transmission 

5. 1.4. 2.1 

Metric 

Visual  Human-Initiated  Communication 

Ml 

Percent 

Of  visual  messages  UTACC  received 

M2 

Percent 

Of  visual  messages  UTACC  received  correctly 

M3 

Percent 

Of  visual  messages  UTACC  received  correctly  on  first  transmission 

M4 

Percent 

Of  visual  messages  UTACC  relayed  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

5.1.4.2.2 

Metric 

Audible  Human-Initiated  Communication 

Ml 

Percent 

Of  audible  messages  UTACC  received 

M2 

Percent 

Of  audible  messages  UTACC  received  correctly 

M3 

Percent 

Of  audible  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent 

Of  audible  messages  UTACC  relays  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

5.1.4.2.3 

Metric 

Electromagnetic  Human-Initiated  Communication 

Ml 

Percent 

Of  electromagnetic  messages  UTACC  received 

M2 

Percent 

Of  electromagnetic  messages  UTACC  received  correctly 

M3 

Percent 

Of  electromagnetic  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent 

Of  electromagnetic  messages  UTACC  relays  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 

5.1.4.2.4 

Metric 

Haptic  Human-Initiated  Communication 

Ml 

Percent 

Of  haptic  messages  UTACC  received 

M2 

Percent 

Of  haptic  messages  UTACC  received  correctly 

M3 

Percent 

Of  haptic  messages  UTACC  receives  correctly  on  first  transmission 

M4 

Percent 

Of  haptic  messages  UTACC  relays  correctly 

M5 

Time 

For  UTACC  to  relay  correctly  from  time  of  initial  transmission 

M6 

Percent 

Of  UTACC  relayed  messages  understood  by  human 

M7 

Distance 

Between  human  and  UTACC  during  messaging 
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APPENDIX  D.  MCT  5.1.4.3  EXPLICIT  ROBOT-HUMAN 
COMMUNICATION  AND  SUB-TASKS 


This  shows  the  highest  level  breakdown  of  MCT  5. 1.4. 3. 


5. 1.4.3 

Metric  Explicit  Robot-Initiated  Communication 

Ml 

Percent  Of  successful  robot-initiated  messages 

M2 

Percent  Of  successful  robot-initiated  messages  on  first  transmission 

M3 

Percent  Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages 

M4 

Percent  Of  modalities  able  to  be  used  in  acheiving  successful  robot-initiated  messages  on  first  transmission 

5. 1.4.3. 1 

Metric  Visual  Robot-Initiated  Communication 

Ml 

Percent  Of  visual  messages  human  received 

M2 

Percent  Of  visual  messages  human  received  correctly 

M3 

Percent  Of  visual  messages  human  received  correctly  on  first  transmission 

M4 

Percent  Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent  Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time  For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance  Between  human  and  UTACC  during  messaging 

5. 1.4.3. 2 

Metric  Audible  Robot-Initiated  Communication 

Ml 

Percent  Of  audible  messages  human  received 

M2 

Percent  Of  audible  messages  human  received  correctly 

M3 

Percent  Of  audible  messages  human  received  correctly  on  first  transmission 

M4 

Percent  Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent  Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time  For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance  Between  human  and  UTACC  during  messaging 

5.1.4.3.3 

Metric  Electromagnetic  Robot-Initiated  Communication 

Ml 

Percent  Of  electromagnetic  messages  human  received 

M2 

Percent  Of  electromagnetic  messages  human  received  correctly 

M3 

Percent  Of  electromagnetic  messages  human  received  correctly  on  first  transmission 

M4 

Percent  Of  human  acknowledgements  understood  by  UTACC 

M5 

Percent  Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M6 

Time  For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M7 

Distance  Between  human  and  UTACC  during  messaging 

5.1.4.3.4 

Metric  Haptic  Robot-Initiated  Communication 

Ml 

Percent  Of  haptic  messages  human  cognitively  received 

M2 

Percent  Of  haptic  messages  human  cognitively  received  on  first  transmission 

M3 

Percent  Of  human  acknowledgements  understood  by  UTACC 

M4 

Percent  Of  human  acknowledgements  understood  by  UTACC  on  first  acknowledgement 

M5 

Time  For  human  to  acknowledge  correctly  from  time  of  initial  transmission 

M6 

Distance  Between  human  and  UTACC  during  messaging 
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